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Abstract 



We consider a "length-biased" shift-dependent information measure, related to the 
differential entropy in which higher weight is assigned to large values of observed ran- 
dom variables. This allows us to introduce the notions of "weighted residual entropy" 
and "weighted past entropy" , that are suitable to describe dynamic information of ran- 
dom lifetimes, in analogy with the entropies of residual and past lifetimes introduced in 
[m and [B], respectively. The obtained results include their behaviors under monotonic 
transformations. 

AMS Classification: 62N05, 62B10 
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1 Introduction 

It is well known that knowledge and use of schemes for information coding and transmission 
play a relevant role in understanding and modeling certain aspects of biological systems 
features, such as neuronal activity. Since the pioneering contributions by Shannon [20] 
and Wiener [55] numerous efforts have been devoted to enrich and extend the underlying 
information theory. Various measures of uncertainty introduced in the past have been 
recently invoked in order to deal with information in the context of theoretical neurobiology 
(see, for instance, Johnson and Glantz [T7]). In addition, recent articles have thoroughly 
explored the use of information measures for absolutely continuous non-negative random 
variables, that appear to be suitable to describe random lifetimes (see [13] and [S], for 
instance). Here, wide use is made of Shannon entropy, that is also applied to residual and 
past Hfetimes (cf., for instance, and [5]). However, use of this type of entropy has the 
drawback of being position-free. In other terms, such an information measure does not 
take into account the values of the random variable but only its probability density. As a 
consequence, a random variable X possesses the same Shannon entropy as X -I- &, for any 

6 e K. 

To come up with a mathematical tool whose properties are similar to those of Shannon 
entropy, however without being position-free, we introduce the notions of "weighted residual 
entropy" and "weighted past entropy" . They are finalized to describe dynamic information 
of random lifetimes. In Section 2 we provide some basic notions on weighted entropy, 

*Paper appeared in Sci. Math. Jpn. 64 (2006), no. 2, 255-266. 
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complemented by some examples. Section 3 is devoted to the presentation of properties 
and of results on weighted residual and past entropies, whose behaviors under monotonic 
transformations are studied in Section 4. 

Throughout this paper, the terms "increasing" and "decreasing" are used in non-strict 
sense. Furthermore, we shall adopt the following notations: 

X: an absolutely continuous non-negative honest random variable (representing for in- 
stance the random lifetime of a system or of a living organism, or the interspike 
intervals in a model of neuronal activity); 

f{x): the probability density function (pdf) of X; 

S = (0, h'), with v < +oo: the support of f{x); 

F{x) = Pi-{X < x): the cumulative distribution function of X; 

F{x) = 1 — F{x): the survival function of X; 

X{x) — f{x)/F{x): the hazard function, or failure rate, of X; 

t{x) — f{x)/F{x): the reversed hazard rate function of X] 

[Z\B]: any random variable whose distribution is identical to the conditional distribution 
of Z given B. 

2 Weighted entropy 

The differential entropy of X, or Shannon information measure, is defined by 

(1) H:^- E[log /(X)] - - / fix) log fix) dx, 

Jo 

where here "log" means natural logarithm. The integrand function on the right-hand-side of 
P]) depends on x only via fix), thus making H shift-independent. Hence, H stays unchanged 
if, for instance, X is uniformly distributed in (a, b) or ia+h, b+h), whatever /i G M. However, 
in certain applied contexts, such as reliability or mathematical neurobiology, it is desirable 
to deal with shift-dependent information measures. Indeed, knowing that a device fails to 
operate, or a neuron to release spikes in a given time-interval, yields a relevantly different 
information from the case when such an event occurs in a different equally wide interval. 
In some cases we are thus led to resort to a shift-dependent information measure that, for 
instance, assigns different measures to such distributions. 

In agreement with Belis and Guia§u [3] and Guia§u [16], in this paper we shall refer to 
the following notion of weighted entropy: 

(2) ■.= -E[X\ogfiX)] = - xfix) log fix) dx, 

Ja 

or equivalently: 

H^ = - dy fix) log/(x)dx. 

Jo Jy 

Recalling Taneja [21] we point out that the occurrence of an event removes a double uncer- 
tainty: a qualitative uncertainty related to its probability of occurrence, and a quantitative 
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uncertainty concerning its value or its usefulness. The factor x, in the integral on the 
right-hand-side of ^ , may be viewed as a weight linearly emphasizing the occurrence of the 
event {X = x}. This yields a "length biased" shift-dependent information measure assigning 
greater importance to larger values of X . The use of weighted entropy ^ is also motivated 
by the need, arising in various communication and transmission problems, of expressing the 
"usefulness" of events by means of an information measure given by iJ™ = 'E\I{u{X), /(X))], 
where I{u{x), ,f{x)) satisfies the following properties (see Belis and Guia§u ^3j): 



I{uix)J{x)gix)) ^ I{u{x), fix)) + I{u{x),g{x)), I{Xu{x)J{x))=XI{u{x)J{x)). 



The relevance of weighted entropies as a measure of the average amount of valuable or useful 
information provided by a source has also been emphasized by Longo [rS|. 

The following are examples of pairs of distributions that possess same differential en- 
tropies but unequal weighted entropies. 

Example 2.1 Let X and Y be random variables with densities 



Their differential entropies are identical (see Example 1.1 of [6], where a misprint has to be 
noticed): 



Hence, the expected uncertainties for fx and fy on the predictability of the outcomes of X 
and Y are identical. Instead, H"^ < Hy, since 



Hence, even though Hx = Hy, the expected weighted uncertainty contained in fx on the 
predictability of the outcome of X is smaller than that of fy on the predictability of the 
outcome of Y. 

Example 2.2 Consider the piece- wise constant pdf 




Hx=Hy = --\0g2. 




X 2x log 2xdx — - 






Its differential entropy is Hx 



J2k=i '^k logCfc while its weighted entropy is 



(3) 




k=l 



Note that any new density obtained by permutation of ci, C2, . . . , c„ has the same entropy 
whereas its weighted entropy is in general different from ([3]). 



Hereafter we recall some properties of differential entropy ([T]) : 
(i) for a, 6 > it is Hax+b = Hx + log a. 
(a) li X and Y are independent, then 



H(X,Y) — Hx + Hy 



where 



p+ca r + ca 

Hix,Y) ■■= - E[log f{X, Y)]^- / fix, y) log f{x, y) dx dy 

Jo Jo 



is the bidimensional version of ([T]). 

Proposition 2.1, whose proof is omitted being straightforward, shows the corresponding 
properties of H"^ similar to (i) and (ii), but the essential differences emphasize the role of 
the mean value in the evaluation of the weighted entropy. 

Proposition 2.1 The following statements hold: 

(i) for a,b>0 itis H^x+b = a[H^ + E(X) logo] + b{Hx + log a). 

(ii) if X and Y are independent, then 

H}":^^y^=E{Y)H^+EiX)H^, 
where Hf^ y^ := -^[XY log f{X,Y)]. 

Let us now evaluate the weighted entropy of some random variables. 
Example 2.3 (a) X is exponentially distributed with parameter A > 0. Then, 

(4) H^ = - xXe-^^ log (Ae~^ dx ^ ^ ~ ^ . 
(h) X is uniformly distributed over [a,b]. Then 

(5) 1 iog^dx = ^log(6-a). 

— a b ~ a 2 

It is interesting to note that in this case the weighted entropy can be expressed as the 
product 

(6) i/"' = E(X) H. 
(c) X is Gamma distributed with parameters a and (3: 

2,Q-lg-2;//3 



if X > 0, 



f{x) = \ /3«r(a) 

otherwise. 



There holds: 



+00 ^a^ — -0 



= a(3 log[/3"r(a)] - a{a - l)/3 { log/3 + ?/'o(a + 1)} + a{a + l)/3, 

where 

(7) ^„(.,:=iL,<.gr(.,.rM 



and where we have used the foUowing identities, with a > —1 and /3 > 0: 



+ 00 ^ 1 ^ -1 — Q 







x°'e-'^ dx^ [-] r(l + a), 



J x"e-t logxdx = f -j r(l + a){log/3 + Vo(l + a)}. 
(d) X is Beta distributed with parameters a and (3: 

ft \ ) — if < a; < 1, 

I otherwise, 



with 
Then, 



B{a,(3)= [ x^-^l ~ xf-^ dx ^ 
Jo 



r{a + (3) ' 



logi3(a,/3)r(« + l)r(/3) ia-l)T{a + l)riP) , , ^ j_ « ^ 1 U 

- Rr^rr {MPMc. + 1) - ar^V-oCa + /3 + 1)}, 

i?(a,/j) r(Q; + /3 + 1) ' 

where "i/'ol^^) is defined in ([7|. 

Remark 2.1 We point out that different distributions may have identical weighted en- 
tropies. For instance, if X is uniformly distributed over [0, 1] and Y is exponentially dis- 
tributed with mean e^^, then from ([4]) and ([5]) we have 

H^=H^ ^ 0. 

Another example is the following: if X is uniformly distributed in [0, 2] and Y is exponen- 
tially distributed with parameter A — 1.93389 (such that log A + A log 2 = 2), then from (|4]) 
and (O one obtains: 

=H^ = \og2. 

Remark 2.2 There exist random variables having negative arbitrarily large weighted en- 
tropy. For instance, if X is uniformly distributed over [a, 6], with > 0, then from ([5]) we 
have 

hm H'" = hm log(6 - a) ^ -00. 

a — a — >b 2 

Remark 2.3 Notice that in general can be either larger or smaller than H. For in- 
stance, if X is uniformly distributed over [a, b], from ^ it follows H'^ > H when E(-'^) > 1, 
and H"" < H when E(X) < 1. 

Remark 2.4 If the support of X is [0, i'], with i' finite, the following upper bound for the 
weighted entropy of X holds: 

(8) H'" < A*log where n = E(X) e [0, ly]. 
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This follows via the continuous version of Jensen's inequality. The maximum of 

2 

//log is attained at /j, = where 

{— if < 2e , ^ \ TT iiv <2e, 

2e ' ■with ^(a<m) = < 2e ^ 

if 1/ > 2e, '^log— if > 2e. 

If, in particular, X is uniformly distributed over [0, i^], then ([8]) holds with the equal sign 
since iJ™ = logt^ and ^ = i//2. 

3 Weighted residual and past entropies 

The residual entropy at time t of a random lifetime X was introduced by Ebrahimi f5] and 
defined as: 

' Jt F{t) ^F{t) 
= logF(i)-=^/ /(a;)log/(x)dx 

(9) =i___y^ /(a:)logA(x)da:, 

for all t G S. We note that is the differential entropy of the residual lifetime of X at 
time t, i.e. of [X | X > i]. Various result on H{t) have been the object of recent researches 
(see [I], [1], [in], [TT], [H]). The past entropy at time t of X is defined by Di Crescenzo and 
Longobardi [5] as: 

(10) 77w.-£||>log/|d., t.S. 

where f{x)/F{t), < x < t, is the pdf of the past lifetime | X < t]. Hence, given for 
instance that an item has been found failing at time t, Hit) measures the uncertainty about 
its past life. We remark that Hit) can also be viewed as the entropy of the inactivity time 
[t-X\X<t]. 

Various dynamic information functions have been recently introduced to measure dis- 
crepancies between residual lifetime distributions [T3| and between past lifetime distributions 
[3, as well as to measure dependence between two residual lifetimes [5]. In order to intro- 
duce new shift-dependent dynamic information measures, we now make use of ^ to define 
two weighted entropies for residual lifetimes and past lifetimes that are the weighted version 
of entropies ([9]) and pO)) . 

Definition 3.1 For all t £ S, 

(i) the weighted residual entropy at time t of a random lifetime X is the differential weighted 
entropy of [X \ X > t] : 

(11) ff- (i) :- - a; £^ log £^ da:; 



Fit) -Fit) 

(a) the weighted past entropy at time t of a random lifetime X is the differential weighted 
entropy of[X\X<t] : 

(12) 7r'«,:=-/.Z|,og|||.. 



We notice that 



lim = lim H {t) = H'' 

t— ► + CX3 t^0 + 



In addition, due to (jlip . the weighted residual entropy can be rewritten as 
(13) H-{t) = -=-J^ ^/(^)log/(^)da; + ^-iiy^ a;/(x)da;. 



The second integral in p3)) can be calculated by noting that 



(14) 



+ 00 



+ 00 



Fiy) dy. 



xf{x) dx ^tF(t) + 
Furthermore, an alternative way of writing H^{t) is the following: 



, fix) , fix) , 
dx / 4=-^ log 4=-^ dw 



dy 



fix) , fix) , 
log _, . da; 



It Fit) ''Fit) 
Recalling we thus obtain: 



+ 00 



dy 



-\-oo 



Fit) ''Fit) 



■ dx. 



(15) 

From (flS)) it easily follows 
(16) 



/ + 00 
Hiy) 



^W^t)^tl^Hit). 



dy. 



In analogy with Theorem 1 of Belzunce et al. [5j, the following characterization result 
holds. 

Theorem 3.1 If X has an absolutely continuous distribution function Fit) and if Hit) is 
increasing for all t Cz S, then H^it) uniquely determines Fit). 

Proof. From ((T3]) and HI]) we have: 

f + OC 



X fix) log fix) dx = log Fit) 
Differentiating both sides we obtain: 
-tfit) log fit) = -tfit) logFit)~tfit)- 



tFit) 



Fiy)dy 



-H"" it) Fit). 



Fit) 



Fiy)dy + fit)H-it)~Fit)-H-it). 



Hence, due to (IT51) and recalling the hazard function A(t) = fit) /Fit), it follows: 



iA(t)[l -logA(t)] = A(i)iJ'"(t) - Xit) 



Fiy) . . d 



— dy~t—Hit). 
Fit) ^ dt ' 



Then, for any fixed t ^ S, A(t) is a positive solution of equation gix) = 0, where 

Fiy) 



(17) 



gix) := X 



til-\ogx)-H'"it) + 



Fit) 



dy 



Note that lim g(x) — —oo and g(0) — t-^HU) > 0. Furthermore, from (fTTll we have: 



it) 



so that 4-g{x) = if and only if 



X = exp 



Therefore, 5(2;) = has a unique positive solution so that A(t), and hence F(t), is uniquely 
determined by H^{t) under assumption -^H^t) > 0. This concludes the proof. ■ 

Remark 3.1 We note that, by virtue of p^ . if H{t) is increasing [decreasing], then also 
H'^{t) is increasing [decreasing]. 

In order to attain a decomposition of the weighted entropy, similar to that given in 
Proposition 2.1 of Di Crescenzo and Longobardi [5], for a random lifetime X possessing 
finite mean E(-^) we recall that the length-biased distribution function and the length-biased 
survival function are defined respectively as 

1 /■* 1 r + oc 

(18) F*it) = -^j^xfix)dx, F*it) = -^j^ xf{x)dx, t&S. 

These functions characterize weighted distributions that arise in sampling procedures where 
the sampling probabilities are proportional to sample values. (See Section 3 of Belzunce et 
al. [5] for some results on uncertainty in length-biased distributions. Moreover, see Navarro 
et al. [19], Bartoszewicz and Skolimowska ^ and references therein for characterizations 
involving weighted distributions). 

Theorem 3.2 For a random lifetime X having finite mean E(^), for all t Cz S the weighted 
entropy can be expressed as follows: 

H"' = E{X) {~F*{t) logF{t)-T{t) \ogF{t)}+F{t)Tr{t)+F{t)H''{t). 

Proof. Recalling Eqs. (O, (HJ) and ((HI) we have: 

= -F{t) j\^-^^\ogf{x)dx-F{t) j^^ x!^^\ogf{x)dx 

= -logF(t) j X f {x)dx -log F{t) J xf{x)dx + F{t)H'"{t)+F{t)H'"{t). 

The proof then follows from (fT8|) . ■ 

In order to provide a lower bound for the weighted residual entropy of a random lifetime 
X, let us introduce the following conditional mean value: 

(19) d{t):=E{X\X>t) = =- xf{x)dx^t+=- F{x)dx, teS. 

t [t) Jt t [t) Jt 



Theorem 3.3 // the hazard function X{t) is decreasing in t E S, then 
(20) H^'it) > ^S{t) logA(i), t E S. 

Proof. From (fTTj) we have: 

1 1 F(x) 

Since log =^ < for x > t and, by assumption, log A(x) < log A(t), there holds: 



1 p + oo 

H^-(t)>-=^j^ xf{x)logX{x)dx 



Fit) 
Fit) 



The proof then follows by recalling ([TI 

In the following example we consider the case of constant hazard function. 

Example 3.1 For an exponential distribution with parameter A > 0, the weighted residual 
entropy is given by 

(21) H-{t) = -j^ ^__iog_^dx = t+--(^t + -jlogA, t>0. 

Recalling that for an exponential r.v. A(t) = A and Sit) = t + j, it is easily seen that (|20|) 
is fulfilled. 

Let us now discuss some properties of the weighted past entropy. From (|12|) we have: 
H'"it) = -J^^j\ fix) log fix) dx + X fix) dx, t G S, 

where, similarly to (fT4|) . it is 



(22) / xfix)dx = tFit)- / Fiy)dy. 

Jo Jo 

Alternatively, 

^'^^-L^''loW)''''W)^' 

(23) = tHit)- f i?(y)dy, 

Jo 

where Hit) is the past entropy given in pO|) . We note that Eq. ([23|l gives: 
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Example 3.2 The weighted past entropy of an exponentiaUy distributed random variable 
with parameter A > is instead given by 



1 



^ ^ e-^* - 2t e-^* - Xt^ e-^* +(\- ^e"^* - t log ^ 



A A VA A / ^ A 



-At 



1 -e- 
for t > 0. 

In order to obtain upper bounds for the weighted past entropy let us now recall the 
definition of mean past lifetime: 

(24) Ai(i) := <t)= J^x^dx = t~^^J^ F{y) dy. 

We incidentally note that this is related to the Bonferroni Curve by ^{t) ~ YjiX) Bp[F{t)] 
(see Giorgi and Crescenzi [T3]). 

Theorem 3.4 (i) For all t € S, it is 

(25) ^^')^^<')'^^^y 

(a) IJ T{t) is decreasing in t £ S, then 

(26) 'H'^{t)<[ a;T(a;)dx-/^(i)[l + logT(t)]. 

^0 



Proof. Eq. (|25[) is an immediate consequence of ([8]). Furthermore, from (|12l) we have: 

~^^^'^^~T{t) ^°ST-{x) dx ~ xf{x) log^;^dx. 

Since r(<) is decreasing in i G 5, we have logT(a;) > logT(t) for < a; < Moreover, 
recalling that log a; < x — 1 for a; > 0, we obtain: 



F{x) 



dx. 



By ([22]) and ^ we finally come to ■ 
Remark 3.2 If X is uniformly distributed on (0, i^), the weighted past entropy is: 

(27) = ^logi, 0<t<v. 

Hence, since /i(i) = t/2, Eq. (|25p is satisfied with the equality sign for all t G (0, v). 

We shall now introduce two new classes of distributions based on monotonicity properties 
of the weighted entropies. 

Definition 3.2 A random lifetime X will he said to have 

(i) decreasing [increasing] weighted uncertainty residual life (DWURL) [IWURL] if and only 
if H'^ (t) is decreasing [increasing] int £ S; 

(ii) decreasing [increasing] weighted uncertainty past life (DWUPL) [IWUPL] if and only if 
H (t) is decreasing [increasing] in t d S. 

Let X be a random variable uniformly distributed on (0, f). Since H^{t) — \og{b—t), 
< t < ly, X is DWURL if and only if < < e, and it can never be IWURL. Moreover, 
from ^ X is DWUPL if and only if < j/ < i, and it can never be IWUPL. Finally, by 
virtue of (PT|) . an exponential distribution with parameter A is DWURL (IWURL) if and 
only if A > e (0 < A < e). 



4 Monotonic transformations 
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In this section we study the weighted residual entropy and the weighted past entropy under 
monotonic transformations. Similarly to Proposition 2.4 of Di Crescenzo and Longobardi 
[6], we have: 

Theorem 4.1 Let Y = (j){X), with (j) strictly monotonic, continuous and differentiable, 
with derivative (p' . Then, for all t ^ S 



{•Mfit) = 



and 



TTyit) = 

(29) 

where 



jjw,4, (0-i(t)) +E{0(X)log0'(X) |X > (t>^^{t)] , 4) strictly increasing 



fj^^'l' +E{0(X)log[-(/)'(X)] \X < , strictly decreasing 



^w,4> +E{(/'(X)log(?!)'(X) \ X < , 4> strictly increasing 

iJ™'-^ {(t)-^{t)) +E{(t){X)log[-4:'iX)]\X > 4:-\t)} , (j) strictly decreasing, 

i^-n.)^-=i-f%(.)/x(.)iog^d., 



H'"''^{t) 



Fxit) 
1 



Fxit) 

'0(x)/x(x)log^dx. 

fxirHy)) 



Fxit) 

Proof. From (fTTj) we have 

H^m- r^ fxi^-\y)) d 

"""^'^-^X ' PiY > t) Ty' 
Let (p be strictly increasing. By setting y = we obtain 

fxix) 



\y) 



log 



PiY > t) 



dy 



r\y) 



H^it) = 



^^""^ Fxi<p-^it)) ^\Fxi<p-^it)) 



dx 



or 



"^^^ J^-Ht)Fxi<p-^it)) Fxicp-Ht)) J^-rit)Fxi^~Ht)) 



r'+oc 

giving the first of ([28| . If is strictly decreasing we similarly obtain: 

'^"^'^ <t>ix)fxix) 



dy. 



dx, 



^^ix) 
dx 



dx, 



Hfit) 



0-1 (+oo) Fx 



-1(0) ^Fxicp-^t)) 
i.e., the second of ((28l) . The proof of ((29)) is analogous 



log 



da; 



cPix) 



dx. 



According to Remark 2.3 of [6 we note that when Y = (piX) is distributed as X (like 
as for certain Pareto and beta- type distributions), Theorem 14. II viclds useful identities that 



allow to express H^it) and Hxit) in terms of i?""'"^ {4>~^it)) or i/""'"^ (^"^(t)), depending 
on the type of monotonicity of (j). 



Remark 4.1 Due to Theorem 14. II for all a > and t > there holds: 
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H:^it) = aH^[^]+Sr-]aloga, 



Kxit) = aH'" ( 1 j + ^ f 1 ) a loga. 



Furthermore, for all 6 > and t > h one has: 

Hl^,{t)=H^{t-b) + hH{t-b), 
Hl^,{t)=Tr{t-b) + bH{t-b). 
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